The present invention generally relates to surveillance systems, and more particularly to a predictive threat detection system that is operative to reanalyze and reinterpret historic image and video data obtained through a sensor network automatically based on current findings.
Effective security against crime and terrorism is a passionate pursuit for nearly all nations. Indeed, the use of surveillance to increase security has becoming increasingly popular for private parties, government agencies, and businesses. It is extremely common in today's society for an individual to look up and realize that she is under the watchful lens of at least one camera while visiting a business establishment or entering a government building. The technology behind this surveillance has exploded in recent years, facilitating a proportionate increase in the use of security surveillance equipment in new locations, and with new purposes in mind.
Security surveillance, although used by various persons and agencies, shares a common goal: to detect potential threats and to protect against these threats. At present, it is not clear that this goal has been achieved with current technology. Indeed, progress toward this goal has been made in moderate steps. An initial step toward this goal was the implementation of surveillance in the form of security guards, i.e. human surveillance. Human surveillance has been used for years to protect life and property; however, it has inherent spacial and temporal limitations. For example, a security guard can only perceive a limited amount of the actual events as they take place, a security guard has limited memory, and often, a security guard does not understand the interrelationship of events, people, or instrumentalities when a threat is present. Thus, a criminal or adversarial force blending into a group may be undetected.
In order to address some of the limitations of human surveillance, electronic surveillance was developed and implemented. In the early 1960's, surveillance technology evolved to include the use of video cameras. See CNN Archive, available at http://archives.cnn.com/2002/LAW/10/21/ctv.cameras/. Early camera systems did not see much success until the advent and promulgation of digital technology in the 1990's, which increased system capacity in memory, speed, and video resolution. See id. Currently, this surveillance allows an individual to view events as they take place (a forward in time, or “forward-time-based” approach) and to record these events for later review. For example, the individual (often a security guard) could monitor multiple closed-circuit cameras for several locations and when necessary, provide physical security enforcement for a given location. Such a system may also be monitored remotely by individual business owners or homeowners over the internet. As may be expected, these systems may vary in complexity—sometimes having multiple cameras and monitoring sensors—depending on the size and importance of the protected area.
As electronic surveillance technology has improved, its use has become more ubiquitous. Governments have begun implementing this technology in large scale to better protect their citizens. For example, England has become known as a world leader in electronic surveillance due to its extraordinary surveillance system. According to the Electronic Privacy Information Center, England has installed over 1.5 million surveillance cameras, which results in the average Londoner being video taped more than 300 times per day. See id. In fact, here in the United States, major cities such as Boston, Chicago, and Baltimore have plans to implement electronic surveillance in order to curtail crime, traffic problems, and adversarial acts. See Jack Levin, Keeping An Eye And A Camera On College Students, The Boston Globe, Feb. 5, 2005, at A11. Indeed, in addition to the reality that electronic surveillance is now here to stay, it is also clear that it will only become more effective in combating crime and terrorism.
Presently, many of the electronic surveillance systems are developing independence from human interaction to monitor and analyze the video data presented on the monitors. Although electronic surveillance is becoming ubiquitous, its reliance on human judgment is problematic due to the limitations and cost of human resources. The developing independence of electronic surveillance seeks to address these shortcomings. In fact, surveillance methods and technologies are being developed that utilize visual tracking and image processing software that do not require human judgment. For example, available technology such as identification and face recognition sensors are capable of measuring the depth and dimensions of faces and places. This technology may be used to identify an ATM user, provide access to an authorized person in restricted areas (and set off an alarm for unauthorized persons), and to monitor three-dimensional rooms, places, and movements of various people and vehicles. See e.g. 3DV Website, available at http://www.3dvsystems.com/solutions/markets.html.
However, similar to the systems previously discussed, these electronic surveillance systems share the inadequacy of human surveillance: they utilize a forward-time-based approach and only archive real-time data for user inspection after the fact. In situations where adversaries operate in an urban environment, by dressing as civilians, driving civilian vehicles, and behaving like civilians, adversaries are able to move about with impunity because even state-of-the-art monitoring and surveillance systems will not detect anything suspicious. When they strike, it is usually a surprise. Worse, when they strike it is already too late to piece together how they set up the attack because there may be no record of the events that lead to the attack, or there is piecemeal information that takes a long time to put together into a cohesive narrative.
While deploying dense sensor networks in an urban environment has become feasible, processing all of the sensor data and tracking all objects in real-time may not be. Predicting the subset of data that will be relevant in the future has proved to be exceedingly difficult, yet without a record of recent events and entity tracking, the utility of these sensor networks is severely limited. Therefore, instead of preventing maleficence, these forward-time-based networks may at best serve to aid a subsequent investigation as to the identification and cause of the maleficence.
Thus, there appear to be several drawbacks to this forward-time-based approach, including: (a) adversaries can disguise themselves to appear and act neutral until they decide to mount an attack, which allows them to utilize the element of surprise and increase their proximity to their objective with little resistance; and (b) even if there are behavioral or physical cues that provide some early warning about the threat, any possibility of discovering where the threat originated is difficult to reconstruct and even possibly lost. The inadequacies of the forward-time-based approach, common to both human and electronic surveillance, has been exposed even more recently through the plainclothes warfare and adversarial attacks seen in recent events.
In particular, forward-time-based surveillance appears to be incapable of preventing deceptive adversarial attacks. Traditional threat assessment in military warfare was a relatively simple task for a soldier with proper training. However, the current trend in military warfare toward terrorism, which is rooted in deception, uses an urban environment to camouflage and execute adversarial operations. Thus, even if real-time recognition of clothing, faces, types of munitions, or a suspicious approaching vehicle were to provide a warning to friendly forces (using a forward-time-based approach), the warning is often too late to prevent an attack. Indeed, although society may sometimes thwart deceptive adversarial attacks through forward-time-based threat assessment, this method is inadequate. Present experience teaches that adversarial forces take advantage of this forward-time-based approach in order to carry out their attacks.
Therefore, there is a need in the art for a threat detection system that is predictive and preventative. There is a need in the art for a threat detection system that is capable of processing and archiving images, video, and other data through a sensor network, and that may analyze this archived data based on current findings. There is a need in the art for a threat detection system that utilizes a short-term memory bank of sensor data to selectively track entities backwards in time, especially one that selectively reserves the use of more effective, but more expensive data processing methods until their use is warranted. Further, there is a need in the art for a threat detection system that is operative to acquire useful information about an adversary, such as home base location, compatriots, and what common strategies and patterns of attack they use. Finally, there is a need in the art for an automated predictive threat detection system that is operative to reanalyze and reinterpret archived and historical data in response to current important events, and to provide a suitable analysis of the discovery and the threat that the discovery poses.